Papers with fine-grained metrics
Measuring and Mitigating Constraint Violations of In-Context Learning for Utterance-to-API Semantic Parsing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | In task-oriented semantic parsing, the system aims to translate users’ utterances in natural language to machine-interpretable programs (API calls) However, Large Language Models (LLMs) are known to hallucinate and therefore pose a formidable challenge in constraining generated content. |
| Approach: | They propose to use large language models to translate user's utterances to machine-interpretable programs (API calls) they identify constraints violations in task-oriented utterrances and define fine-grained metrics that complement traditional ones. |
| Outcome: | The proposed methods reduce constraints violations and improve quality of the generated API calls, but require careful consideration given their implementation complexity and latency. |
World Knowledge for Abstract Meaning Representation Parsing (L18-1)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) parsers are based on annotated graphs, but there is still room for improvement . |
| Approach: | They examine the role played by world knowledge in parsing errors in a state-of-the-art parser . they examine the effects of different types of world knowledge on parsers . |
| Outcome: | The proposed model improves on multiple fine-grained metrics, including a 6% increase in named entity F-score, and provides insight into the potential of world knowledge for future work in Abstract Meaning Representation parsing. |